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tional Bureau of Standards (now NIST) in the areas of precise electrical measurements, electri- 
cal standards, instrumentation, automation, measurement assurance, and other related fields. 
Eicke is author of more than twenty published papers for scientific and technical journals, and 
has served on numerous professional society and NBS committees and participated in stan- 
dards writing. He attended George Washington University where he earned his BS and MS in 
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1.1 Purpose 

Methodologies and techniques acceptable in fulfilling metrology , calibration, and measurement 
process quality requirements for NASA programs are outlined in this publication. The intention 
of this publication is to aid NASA engineers and systems contractors in the design, im- 
plementation, and operation of metrology, calibration, and measurement systems. It is also in- 
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tually simple and straightforward. With the proliferation of computing technology and philo- 
sophical changes occurring in quality management, the field of metrology is undergoing evolu- 
tionary and revolutionary change. Methodology for determining measurement uncertainty is 
becoming extremely complex in terms of system and component error analysis and manipula- 
tion of equations that require a good foundation in mathematics. 
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process, reviews methods for combining errors, and provides the specific steps needed 
to develop and evaluate a measurement process. 
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Appendix F ( Practical Method for Analysis of Uncertainty Propagation) describes an 
evolutionary nontraditional uncertainty analysis methodology that yields unambiguous 
results. The term “practical” suggests that the methodology is usable or relevant to 



Throughout this publication, references are made to “space-based” activities. 
For the purpose of definition, “space-based” includes all activities that are not 
Earth-based, i.e. satellites, humanly-occupied on-orbit platforms, robotic deep- 
space probes, space- and planet-based apparatus, etc. — all are included in the 
term “space-based” as used in this document. 
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catty estimated and controlled within the specified range, and that out-of-control exceptions are 
objectively identified. 



NHB 5300.4(lD-2), “Safety, Reliability, Maintainability and Quality Provisions for the 
Space Shuttle Program” 


NHB 4200.1, “NASA Equipment Management Manual” 
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PRINCIPLE 2 — Measurements are made to support decisions 1 or establish 
facts. If measurement data are not used in a decision, the measurement is un- 
necessary. 



3.2.1 Measurement Requirements Definition Sequence 

Determining measurement process requirements can be viewed as a ten-stage sequence that 
flows down as follows: 


Define the objectives of the mission. What is to be accomplished? What reliabil- 
ity is needed and what confidence levels are sought for decisions to be made 
from the measurement data? 
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Define the calibration measurement parameter values, ranges, uncertainty 
limits, confidence levels, and recalibration time limits (calibration intervals) 
that match measurement system performance requirements to detect and cor- 
rect for systematic errors and/or to control uncertainty growth. 









STAGE 9 — CALIBRATION SYSTEM DESIGNS 
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In the ten-stage flowdown of determining measurement process requirements, two aspects are 
indigenous to the process. They are: the underlying operational requirements and the special 



Show the confidence levels needed for measurement decisions. 




3.2.3 Establishing Measurement Classifications 

Another facet of defining measurement requirements calls for consideration of the relative im- 
portance of all measurement processes involved in a given program or mission. Indicators of 
importance are useful in identifying confidence level requirements on measurement uncer- 
tainties in a program or mission. 
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3.2. 3.2 Difficulty of the Measurement 

The degree of difficulty of each measurement may have a direct effect on it’s cost and quality. 



less than 10 percent (1/10) of the measured parameter tolerance limits. 




sign proper confidence levels is needed to help planner and designer alike and is addressed in 
the next section. 



TABLE 3.1 

Measurement Uncertainty Confidence Level 
Assignments for Measurement Applications and 
Degrees of Difficulty 






3.2.5 Establishing Measurement System Reliability Requirements 

The previous section provided guidance on confidence level assignments for measurement un- 
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Section 3 — MEASUREMENT REQUIREMENTS 




3.2.6 Finalizing Measurement Requirements 

Once the measurement parameters, measurement values, applications, environment, and tol- 
erances (including confidence/reliability limits) have been defined, the final definition of mea- 
surement requirements is nearly complete. 
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Example — Measurement Requirement Definition of a Solar 
Experiment 




'Section UBI 1 Ml ' 



MTBOOT = -Usage Hours per Year/ In R 
Where R = confidence level or measurement reliability. 
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bration system will be periodically rotated to NIST for calibration so fresh lamps, within their 


In this and other similar cases, each nation, including the U.S., has established reference 
standards for a particular quantity. They do not rely on a single international standard. In- 
stead, they conduct periodic intercomparisons and measure the difference between the as- 
maintained standards representing a particular unit (here, the unit of irradiance — 
watt/meter 2 ). 
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3.2.8 Compensating for Difficult Requirements 

It often seems that the most critical and difficult of measurements are the hi 
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applications. Working backward from end-item accuracy requirements enables the quantifica- 
tion of accuracies needed for test system calibration. Working backward from these accuracies 
enables the determination of accuracies needed for calibration of calibrating systems, and so 
on. The method for doing an analysis of this kind is discussed in Section 4 and is presented in 
detail in Appendix C. 
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based calibrations are discouraged for several reasons, such as time, space, weight and prior- 
ity considerations. For those measurement systems whose calibration intervals are estimated 
to be shorter than the mission duration requirement, special in-place calibration or interval 



certainty over time so that the calibration interval matches the time where uncertainty 
growth has reached a point equal to a 10: 1 process before recalibration is due 

Build in redundant and compensating measurement circuitry to improve reliability 

Provide physical adjustment points that are readily accessible without major disas- 
sembly of the equipment — all easily accessible adjustments should be sealed after cali- 


Use alternative or multiple measurement sensors with comparison devices 

Standardize easily accessible interfaces to instrumentation to simplify calibration 

Tighten end-item hardware tolerance requirements to create more conforming hard- 
ware that can tolerate the lowered confidence levels generated by the increasing uncer- 
tainty over time of the measurement process 




So-called self-calibration or self-test systems are useful, but are rarely substi- 
tutes for complete periodic calibrations — they serve mainly as interval ex- 





If these conditions are met, application of the SMPC methods can serve to make payload mea- 
suring systems somewhat self-contained. This subject is covered in detail in Section 6.4 and 
Appendix D. 
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explanations of the effect(s) of hardware changes. Measurement uncertainty monitoring 
during operations must also be based on a known hardware configuration. 
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Figure 4.1 — Overview of the Measurement System Design Process 
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decomposed and used to establish system specifications including anti-alias filter characteris- 
tics, sampling rates, and throughput. 



Figure 4.2 — Example of a Measurement System Design Process. 


It is assumed the measurement requirements have been analyzed to establish measurement 
system specifications and the measurement requirements have been formalized (Section 3). 
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4.2.3 Method of Transduction or Energy Transfer 

The physical process that provides a usable output in response to the specific measurand 
should be identified. For example, when measuring temperature, establish the primary mode 
of heat transfer (conduction, convection, or radiation). 
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gain error should be less than ±0.1% FS (within stated confidence levels and time limits). 
Manufacturer specs are statements of performance. If the manufacturer’s specs will be used as 
references for estimating uncertainties, the instrument user needs to do the necessary cali- 



Next, the environmental limit of the instrument must be determined to identify those contribu- 
tors to other uncertainties that can and cannot be corrected or compensated for. These include 
thermal responses, vibration sensitivity, moisture effects, radiation effects, etc. 
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Evaluating Measurement System Errors 

Understanding, identifying, and quantifying the various error sources is a pre- 
requisite for determining design adequacy and establishing calibration re- 
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Aliased data cannot be distinguished by a computer, nor can aliased data be 
eliminated after it has been sampled. Once A/D conversion is completed, there 
is no way to know from the sampled data whether aliasing has occurred. Even 
if it were possible to know, there is no way to correct the digital data for alias- 
induced errors. 
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4.4. 3. 2 Description and Mechanism of Aliased Data 

Figure 4.5 illustrates three sine waves, each one simultaneously sampled by the A/D con- 



FlGURE 4.5 — SIMULTANEOUS SAMPLING OF THREE SINE WAVES. 
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4.4.7 Data Reduction and Analysis Errors 

Correlation of data reduction methods and the characteristics of the measurand must be an 



To a first-order approximation, the change in the function F, denoted 5F , is related to the 
changes in the measurements M ( , denoted A M { as follows: 
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Since Eq. (4.3) has n unknowns, a unique solution does not exist. Equation (4.3) gives the de- 
signer a mechanism for budgeting uncertainties to each of the n measurements. The examples 



tributed about the mean value. The repeatability of the measurement depends on the precision 
errors. If, at a specific value of the measurement, the bias and precision errors are known, 
they can be combined to establish an estimate of the uncertainty associated with the mea- 
surement. 


Measurement 

Value 





o o o 
5 45*2 
d 3 
c c c 

CD CD CQ 



8 8 8 

CD CD CD 

,aao. 
CO CO CO 

CO 0 ) 0 ) 

U. w. V 
2 2 2 
5 5 5 
o o o 

03 CtJ CO 
=3 =3 3 

c c c 
5 5 5 
§ § § 


o 

CCC 

,o ,o .o 
,55 .55 ,55 

COCOCOCO^OOCO 
CDCDCDCDggg.CD 
S S S ffi CL IX (X 55 


zl M 

§ 8 .f fi 

(/)Co 5 d 


co 

5 5 8 

8 85 

c c ^ 

•£•■§ i 

S.S-i 

0 . 0 . 1 
Q. Q..S 
*C § 


CO CO co 
CD ,C0 .CD 

cS £0 CQ 


55 1 1 1 O 

i$. c f 4 S 1^.8 8 

h“ i8 CD O ^ CD “c: o o 


co co CO ,co 

tilt 

CD CD CD CD 
C C C C 
^ ^ 
c c c c 
.O .O .O ,Q 

•C 5 

CD CD CD CD 
.O .O O .O 

o. aaa 
cl Q. Q- o. 
^ ^ ^ ^ 


c c c 
,o ,o o 
co .55 .55 
o o o 
2 2 2 
CICCU 
v. .y ^ v-. 

o .co o o 
co ^ CO co 

CD g .CD .CD 

(5 cl tS <3 


o c 

z _q>.o 

Q- 5 5 

nil 

to "C ^ 


DC 0 . 0 . 

goo 

£11 

■” CD CQ 


< as 
£ 2>5 

DC gj .C co 

sy ^ 5 s 8 

t § O co 2 

?o^oo 


DC — 

— CD r~ 

> jo •£ 

z ^ <g 

ill o o 


5 - S ^ w 

llfli 

f la 1 a 
U s « a ~ 

CO O *2 - On 

3 « ^ fl V 

<l> « 2 a o 

a -a i ’“fi 
jas-S s 

t> 3 $ & <u 

s I* s 5 s 

k s S §> § 

l> 8 2 ■S 5 

S a a «? 93 

. h ^ ai *>. 

« * <g § -a 

^ ij o S ^ 

1 ^ t; t) K 
iob <2 §* S c 

2 8 8 8 § 


B « S 3 

•3 Jh fW O ^ 

■gfi S »8 
fi i “. £ I S 

.o « -So 


£ a £> b ° 

JB § I & S 

rt P A h h 

& I g § " 

# 1 § 1 1 
r „ S tj ® o 
J B ^ i; 

rt Qj O O f-h 

,H -rH 1 CO -2 <D 

•g 3 ” 2 » 

8 gg » a 

S ^ * 

^ 95 t! « v 

cS >3 « 

8 2 g % s 

SK 8 u 2 § 

533 fil 


* <U 3 
u +s ^ 

<U cfl ’O 
b fl v 


y m g 

is 

§ (N O 


|i | 

4) V g 

b Is t* 
c 2 2 

U 3 53 

T-j ••— l . 

<g a> 8 

fi .5 o 

O fi Jh 
O CD fi 

a; o 
S fi « 

its 

*Sfc 

(U .CD 

^ g ° 

C+H CD un 
° 2 ^ 

2 ^'fl 

s i i 

o S § 
■§ b‘b 

Sbt 

73 ° i 


S ° 2 

b to o 

<l> b 

£ &«2 

H to H 



common units and confidence levels. 










4.5.3 Establishing the Total Bias Estimate 

At a specific measurement value, the various biases listed in Table 4. 1 are established and 
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Section 4- MEASUREMENT SYSTEM DESIGN 



4.5.6 Example — Budgeting Measurement Uncertainty in the 

Design Process 

Consider the requirement to develop a measurement system to measure the velocity of air in a 
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Interpretation of Solution 



4.6.1 Measurement Uncertainty 

Measurement errors are never known exactly. In some instances they may be estimated and 
tolerated or corrected for. In others, they may be simply acknowledged as being present. 
Whether an error is estimated or acknowledged, its existence introduces a certain amount of 
measurement uncertainty. 









Approximations and assumptions incorporated in the measurement methods and pro- 
cedures 
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4.7.1 Temperature Measurement System Equipment Selection 
and Specification Interpretation 

The basic elements comprising the example temperature measurement system are shown in 
the following sketch: 
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For best results, thermocouples should be calibrated before using. They should be calibrated 
at the temperature range of interest to lessen and quantify errors due to variations in materi- 
als and manufacturing. Calibration will allow for careful selection of thermocouples, which 



Often, the measurement uncertainty requirement is impossible to meet. If the requirement had 
been, for example, ±1 °C, 3c, and given the manufacturer’s published data of ±2.2 °C at 20- 
100 °C for a reference thermocouple, segments of a roll of thermocouple wire would have to be 
individually calibrated to find lengths that would reduce the Limits-of-Error to less than ±1 °C. 
If this cannot be accomplished, the measurement uncertainty specification must be relaxed. 
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STEP 3. Identify and quantify intrinsic equipment errors and confidence levels. 

Gain Accuracy: ±0.003% /°C, ±0.05% FS [Given] 
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The specific manner in which uncertainty growth is accounted for depends on 
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Measurement parameter uncertainty growth for the linear drift model is depicted in Figure 6.2 
of Section 6. Other uncertainty growth mechanisms and associated models are described in 
Appendix B. 
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One of several definitions of traceability is: 
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“Calibration Labs” represents many laboratories of varying capabilities and may be multilay- 
ered. They may go directly to NIST or to another calibration laboratory. At each stage, there are 
error sources producing measurement uncertainties propagated to the next level. Also, the 



For differing units representing a quantity, different values for {N} will result. This can be seen 
by considering the measurement of an invariant quantity using two different units. Since the 
quantity is invariant, the following relationship results: 




5.1 .2 Definition of Tolerance, Uncertainty, and Accuracy Ratio 

Following are the definitions of tolerance, uncertainty, and accuracy ratio: 

Tolerance — Tolerance is a condition imposed on a measurement by the designer or other 
agency. Tolerance is defined as “the total permissible variation of a quantity from a designated 
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The notice published in the Federal Register states: 

Under both this act and the Metric Conversion Act of 1975, the “metric system of measure- 
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5.3.3 NIST National Standard Reference Data Program (NSRDP) 



5.4.1 Reciprocal Recognition of National Standards 





reciprocal agreements. Many nations, including the U.S. do many comparisons with no regard 



limits of error relative to national standards. 
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problems; a small or no change shows that the transport process has not affected the 
item. 




co > u 

93 S3 fH 
13 C3 G3 

G u co 
3 C « 

IJS? 


i fli t 

I 6 fc 

3 s s 

CO „ M 
T3 CO _ 

H co U 
ro <U rj 

1 2 S 

| kg 

® G O 

<L> O rt 

a =g S 

§ 2 a 

a s 

V Si h 
!h O O 

4G fc 

G O 43 

o rt 

s 

rt |T3 

a <l) 


G 0 ^ 

S Gi 

I § i 

£ *£ - 
^ H « 


<u o cd 

^ I .2 





tainty determination. When this model is used in conjunction with the calibration model, vari- 
ous error sources can be identified and quantified. Operationally, MAPs rely on the use of a 
check standard to monitor the process continuously. By repeated measurements on the same 
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confidence levels axe required of the process. 
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5.5.3 Regional Measurement Assurance Program (RMAP) 
Transfers 



information is obtained.) Each artifact is measured by each participant, and all results are 
then analyzed. The two artifacts do not need to be identical, but they must evaluate the same 
measuring process. The round robin done by the Kennedy Space Center for voltage, at the 10 
V level, illustrates the idea. 



Youden Plot 
10 Volt Round Robin 
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ence factors, to obtain a value to be used with the standard to perform calibrations. 
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latter. The confidence level may be arbitrarily chosen, but in any error analysis the chosen 
level must be stated. 
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5 . 7.2 Propagation of Uncertainty in the Calibration Chain 

Errors made at higher levels are propagated to the next level. Since the true error cannot be 
measured directly, the uncertainty is the tool by which error estimates are transferred down 
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(2) Many instruments can store corrections in nonvolatile memory. In use, the instrument 
logic handles proper application of the correction to display the correct value. Proce- 
dures for using such features must be unambiguous. Several measurements should be 
taken after calibration to ensure that corrections were properly installed. 


|f|f 

fe » 13> &■ 


fi 9 j 
,3 b S 


v . Z 

O V <D 

rt-j 


d d *n d 
a; co fl £ S 

, s ^>*1 v ~ 

2 B | 
glsil 

<3 ^ « o 
d v & ^ 
V o W rj 

S U O <L> H 

H C Jd -2 

® a 3> * . a 

5 -a % g a 

•rH fH M O CT 

£ *S =3 * a 


2 co -h 
^ cd cd 
-O « fl 

<D rH ^ 
5P- S 03 


S g § £ 

V g **3 rQ 

3 g •§ * 

^ 3 u 
® a xi c3 
3 u '2 £ 

| a 

§ V . w 

w ^ d ^ 
cd O co Q 

w fi -j {/) 


3 C 
dr 1 d 

i§ J 

w cd 
^ ffi 

1*3 

o 8 •a 

a S a 
£ 6 .s 


03 » 13 
0^1) 

$■&! 

^5 U d 

§ b *8 
5 a "0 

CJ <L) 


a Tj e 
5 <u <3 

0 T5 ±j 

1 <L> £ 
O a) H 
^ fi ^ 


% & 

I N § 

,i2 u » ™ 
rf 3 3 D 
C ^ ^ CO 


3 03 CT 1 Cm ■ 

• ae ° 

& ^,3 3 

2 -a o o 

3 3 « 53 

1 6 -O § ' 

§ lit 

Sb co “j s' ' 
v #h cd _d 

-c S T 3 43 




5.9.1 Software Documentation 



• • 


5.9.2 Software Configuration Management 

Configuration management is a critical but often neglected function in small installations and 
projects. 










Although satisfying all three requirements is often difficult, methods and techniques have 
emerged for establishing and adjusting calibration intervals that promote meeting both prod- 
uct assurance and cost control objectives. 
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6.2.2 Measurement Reliability Targets 
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reliability targets, leads to the following objectives of effective calibration interval analysis sys- 
tems: 
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6.2.5 Calibration Interval Elements 
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6.2.5.2 Reliability Modeling 

Uncertainty growth processes are described in terms of mathematical reliability models. Use of 



6. 2. 5. 5 Logistics Analysis 

Logistics considerations must be taken into account to synchronize intervals to achievable 
maintenance schedules. Interval synchronization is also required in setting intervals for TME 
models, such as mainframes and plug-ins that are used together. 


6.2.5. 6 Costs and Benefits 

Operating Costs — Obviously, higher frequencies of calibration (shorter intervals) result in 
higher operational support costs. However, because of uncertainty growth, longer intervals 
lead to higher probabilities of using out-of-tolerance TME for longer periods of time. 

nptprminatinn nf the balance between onerational costs and risks associated with the use of 
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6.2.5.7 Personnel Requirements 



Unfortunately, calibration interval systems are not immune to “improvements' 
made by personnel unfamiliar with system theory and operation. 
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attribute-to-TME parameter tolerance ratio is large. This ratio is traditionally referred to as the 
TME-to-end-item “accuracy ratio.” 
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Section 6 - CALIBRATION INTERVALS 




Measurement Reliability Modeling 

• Represent the time series with mathematical reliability models 

• Construct the likelihood functions 

• Obtain maximum likelihood estimates of reliability model coefficients 
(analyze the time series to infer the uncertainty growth process) 




6.3.5 Measurement Reliability Modeling 

A number of uncertainty growth processes are possible. Each process corresponds to a par- 
ticular mathematical description or model. Each model consists of a mathematical form char- 
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An interval adjustment may either shorten or lengthen an interval. In the discussion that fol- 
lows, both adjustments are treated as being equal, with no distinction made between the QA 
approval requirements for, or advisability of, each. The discussion merely assumes that any 
interval adjustment (longer or shorter) is based on supporting data and that the adjustment is 
made in such a way as to strive toward meeting specified reliability targets. There are three 
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Dog/Gem Identification — The requirements for valid calibration intervals, based on analysis of 
data sufficient for accurate measurement reliability modeling, and the need for responsiveness 
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parts performance. (OOTR is the inverse of the mean-time-between-out-of-tolerances 
[MTBOOT] referred to earlier.) The OOTR can be applied in mathematical reliability models, as 
if it were obtained from calibration history data, to determine an initial calibration interval es- 
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6.3.8 Equipment Adjustment Considerations 

During calibration, decisions are made whether to adjust or correct parameters under test. 
Typically, TME is adjusted to match the values of its calibrating TME. Three categories of ad- 
justment practice are encountered: 
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Over well-behaved portions of a parameter’s operating curve, such corrections are entirely 
equivalent to physical adjustments. However, if parameter values drift or otherwise transition 
to unstable portions of their respective operating curves, software corrections alone are not 
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Establishment of an appropriate measurement reliability target is a multifaceted process. The 
major points in establishing a measurement reliability target follow: 


TME measurement reliability is a measure of TME uncertainty 

TME uncertainty is a major contributor to the uncertainty of the end-item’s calibration 

The uncertainty of the end-item’s calibration impacts the uncertainty of the measure- 
ments made with the end-item 

Measurement uncertainties impact end-item usefulness. 
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STEP 2. Determine TME parameter tolerances that correspond to acceptable 
test process uncertainty. 
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Visibility of the uncertainty growth process for each TME parameter is obtained by sampling 
the time series that reflects this process. Data can be collected through recording the results of 
periodic calibrations for TME deployed in ground-based applications or can be accumulated 
through controlled experimentation during TME design and development. For the latter, care 
must be exercised to match the experimental conditions with those anticipated in actual us- 
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116 Section 6 — CALIBRATION INTERVALS 




STEP 7. Compute calibration intervals commensurate with appropriate mea- 
surement reliability targets. 
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ences and/or terrestrial and/or astronomical references or (2) to tolerate extended periods 
without calibration or testing. Using the measurement reliability modeling methods described 
in Appendix B, designs can be evaluated in terms of how well these objectives will be met. The 
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STEP 2. Determining TME parameter tolerances that correspond to acceptable 
test process uncertainty. 



cording to mission requirements. 


For extended deployment applications, measurement reliability targets should constitute de- 
sign goals for each TME and end-item parameter. Ordinarily, this practice is not followed. For 
example, a specification for detector sensitivity might read something like the following: 
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have been routinely reported by TME calibrating organizations. Second, stating tolerance limi ts 
in such a way that they cany with them a zero expectation for attribute or parameter out-of- 
tolerance is suboptimal for measurement uncertainty management. There are three problems 
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tributes of TME parameters of interest. 


In the design/ development phase of a system’s life cycle, visibility of the uncer- 
tainty growth process for each attribute or parameter is obtained in two stages. 
The first, measurement reliability network modeling and simulation, is appli- 
cable to the design phase. The second stage, empirical uncertainty growth 
modeling, is applicable to the preproduction or prototype phase. 
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may consist of an event tree network integrating the stabilities of individual components into a 
composite description of the entire board. In cases where boards are relatively simple, the 
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Similar item method — Extrapolation from the known measurement reliability of existing at- 
tributes performing similar functions under similar conditions and employing similar design 
approaches to those intended for the attribute or parameter of interest. 
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Functional reliability network modeling is covered extensively in the reliability literature. Al- 
though many of the same principles apply, measurement reliability modeling is not qovered in 
any known body of established literature. However, such modeling is often performed by TME 
designers, as indicated by published equipment specifications found in TME catalogs and user 
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Statistical Measurement Process Control 
(SMPC) Methods 
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In traditional SPC applications, the monitoring of testing or calibrating processes is done by 
using process control limits. Process control limits consist of performance specifications ex- 
panded to include measurement process uncertainty contributions. These contributions are 
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FIGURE 6.3 — PROFICIENCY AUDIT EXAMPLE. Three instruments measure an unknown 
value. This value may be external to all three instruments or generated by one or more of 
them. Instrument 1 is arbitrarily labeled the UUT. Instruments 2 and 3 are employed as 





TABLE 6.5 

Proficiency Audit Results Arranged for SPC Analysis 


X \ 








CO 

I 

»a 


CM 

8 

»a 


i 

£ 

=> 


© © © >n 


SO 

o 

OS 

in 

Jl 

11 

II 

II 

X. 



X 


o o o 

r— I t-H i—I 

II II II 


in 

i— > i 

SO 

On 

1 

n 

II 

II 



^ <N 

X 


in 

O 'O I I 

II II II II II II II II 

►4 s -T ^ >? *r ^ 


C co 

S «- 

§■5 s 

o 
G 


<u 

5 


b 

o 

15 


<u 

M CJ 

;> 5 
« 2 

§1 


« -ti 

© e 

E &> 
Jj -3 
nS Os 

a « 

5 a 


Odd 

^ ^ -PH 1 

>> V <D 

* >, -B 


•g - eg 

S 3 B 


1 q 


co 


& 

o 

u 

P. 


P -Hj 
cj rt 

p 

oS 


1 

3 


* SI 


C | 

dj 


s 

£ 

(0 

c 


V 


4J <u CO 

5 5 5 1 


W '♦—i 

^ & s 

> P CO 


-9 'S 

OS « 

•Q -S 

r 


kT v 

x 3 
£ 1 
g S 
a ts 

IS 

CO ■— J 
03 *- J 
CJ o 

as* 

4 . P T) 

° ts g 

” £>X 

rt £ to 

TSS+j 

s-s g 
Sf a 
■8 


CS 


u . 

o* b 


<u 


0) 

c 5 
2 2 
mj g „ . 

?1| 

^3 g 

! ■g 

** 95 P 

^ G ^ 

H - .2 -d 

052, 8 

.S ^ fc 

* “T P 

3 o 

O <w o 
(0 0,0 


td .2 
5 s? 


§ 

£ 

P 

CO 

o3 

CJ 

6 


P 2 
C o 

6 °« a 


<0 <D ^ 

© <2 T3 
-M ° G 

S 3 3 

!» fi n 
8 1; o 
o 

^ oj P 
£> G ^ 

w rj (U 

a £ I 



os 

I 

II 


x 


i 

i 

Oh 

CJ 

CJ 

I 

CD 


§ 

CO 

2 

*& 


CD 

6 * 

W 


T3 

ft 

(0 

B 

Jf 

*co 

D . 

^ M 

N«+i 

X § 

5i 

^ § 
<« -5 

T3 

ajS 

C-t-H 

a 

rP 0 s 
53 05 
tdD 05 

P ^ 

•P O 

S <u 

p 

I 

co 

a; 


T3 

§ 





FIGURE 6.4 — EXCHANGING UUT AND TME ROLES. Instruments 1 and 2 of the proficiency 
audit example exchange roles as UUT and TME 1, respectively. This role swapping is done 
to estimate instrument 2 in-tolerance probability. 






So, as to the proficiency audit question “who’s out-of-tolerance?” the answer is that instru- 
ment 1 has an estimated bias of -7 and an in-tolerance probability of 77%, instrument 2 has 
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” by automatically subtracting the value /? from subsequent monitoring-system 



Analyzing Measurement Decision Risk 





Until recently, establishing acceptable levels of error and uncertainty has been a simple pro- 
cess in which nominal standards of high accuracy between verifying and subject units were 
maintained. Historically, relative accuracies have been such that measurement system uncer- 
tainties were required to be ten percent or less of end-item or product tolerances, and calibrat- 
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through (false accepts). Likewise, because of measurement process uncertainties, some in- 
tolerance attributes are perceived as out-of-tolerance and are rejected (false rejects). 
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Performance Limit Limit which bounds attribute values 

corresponding to acceptable performance 
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6.6.2 Applicability and Responsibility 



The benefits from effective measurement decision risk management can be considerable. 
Gaining these benefits can, however, call for substantial investments. These include invest- 
ments in management energy, data recording and storage, data-processing capability, and 
personnel. 
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that future needs for measurement decision risk management will exceed those of the present 
day. As the costs of technology development and maintenance continue to spiral upward and 
performance criteria continue to become increasingly stringent, it may be assumed that the 
need for effective measurement decision risk management will become an accepted fact of 
twenty-first century life. 
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pass, it was decided that a 3a (99.73%) level should be targeted for the test system and a 2 <7 
level targeted for the calibration system. These and other specifications are summarized in 
Table 6.7, Solar Experiment Specifications; Table 6.8, Solar Experiment End-Item (Prime Sys- 
tem) Information; Table 6.9, Solar Experiment Test System Information; and Table 6.10, Solar 
Experiment Cal System Information. 
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Quantity of Cal Systems 2 

Acquisition cost of one Cal System (parameter) $90,000 

Spares coverage desired 100 % 
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TABLE 6.11 

Solar Experiment — Test & Cal Analysis Results 
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downtime resulting from calibration. Comparison of costs and risks with baseline (original) 
figures is particularly revealing. 

By using the methodology described in Appendix C to analyze end-item attribute support re- 
quirements in terms of effect on mission performance, it can be seen that considerable savings 
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Annual Acceptance Cost 414 414 3 

Total Annual Cost 8,604 9,853 49,049 

Spares Acquisition Cost 12,877 14,093 259,145 
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next calibration. Specifically, the reader should review calibration control provisions of NHB 
5300.4(1 B), paragraph 1B905; NHB 5300.4(1C), paragraph 1C310(4); and NHB 5300.4(lD-2), 
paragraph 1D507(6). 
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7.1 .3.2 Traceability After Equipment Failure 

Equipment failure causes significant opportunity for traceability loss. Typically, traceability 
may be lost because of substitution of uncalibrated devices (e.g., sensors and instruments) 
into the measurement system to continue data-acquisition operations. During the period that 
uncalibrated devices are used, measurement traceability is lost and should be explicitly stated 
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The goal of maintenance is to assure there will be no breakdowns of measurement systems 
and that they can continue to operate safely. Typically, the cost to maintain is traded off 
against the cost to suffer measurement breakdowns and increased safety risks. 
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Maintenance programs will include procedures that ensure: 


(1) Identification and estimation of maintenance requirements. 
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mechanisms, and devices whose failure could be avoided by detecting diminished capability, 
fluid loss, dirt and grease accumulation, environmental stresses, and wear. Also, equipment 
use should be reviewed to find out the experience level of users, their opinions regarding its 



7. 2. 3. 3 Designing Maintenance Procedures 
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Maintenance Interval analysis should stand alone and be based on mathematical and statisti- 
cal correlation of historical failure data that focus on types of maintenance done, time between 
maintenance, failed components/parts, and time between failures. From these data, MTBF 



and even more important when the repairs and calibration are done in the operational envi- 
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Classification of Waiver/Deviation Requests 
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terize system performance, but can, at least, be measured at reasonable difficulty 
levels. 
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NOTE: The following definitions annotated <vim) were prepared by a joint working 
group consisting of experts appointed by International Bureau of Weights and 
Measures (BIPM), International Electrotechnical Commission (IEC), Interna- 
tional Organization for Standardization (ISO), and International Organization of 



surand. NOTE — The use of the term “precision” for “accuracy” should be avoided. 

ACCURACY RATIO — The ratio of performance tolerance limits to measurement uncertainty. 

ADJUSTMENT — The operation intended to bring a measuring instrument into a state of per- 
formance and freedom from bias suitable for its use. (vim) 
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dards or target. Any variations in environmental conditions, calibration procedures, or data 
correction algorithms between the instruments must be accounted for in the assessment. 

CROSSTALK — Signal interference between measurement channels usually due to coupling 
between channels in some element, e.g., power supplies, adjacent cables, and adjacent teleme- 
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DYNAMIC MEASUREMENT — The determination of the instantaneous value of a quantity and, 
where appropriate, its variation with time. NOTE — The qualifier “dynamic” applies to the mea- 
surand and not to the method of measurement, (vim) 



INDICATING (measuring) INSTRUMENT — A measuring instrument that displays the value of a 
measurand or a related value, examples: (a) analog voltmeter; (b) digital voltmeter; and (c) mi- 
crometer. (VIM) 
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INFLUENCE QUANTITY — A quantity that is not the subject of the measurement but which in- 
fluences the value of the measurand or the indication of the measuring instrument, examples: 
(a) ambient temperature; (b) frequency of an alternating measured voltage, (vimi 



CO 

2 


' d ' 

d § £ 

, CO T 3 * 
1 p V T 3 


3 CO ^ 

co cd .Jij 

s a © 




•r-i is m 

(U C 7 1 4; 

S "2 £ 

« 3 I 
I S H 

_ u Z 

9 g w 

< H 1U 

CC 3 CC 

3 <L> 3 

0 ) 35 (0 

< t» < 

IU oj IU 

s S s 


}) u « 

5 si 


<u <u 

■Ifl 

v, 2 a 

o ti b 

pH ^ 

S y co 

<D G 4 -J . 

^ Eo* 2 $ .5 

cd «, £ 2 

> 3 3 u B 

G S > o 

: -a a s a 

| £ 3 2 

^5 1 § 

'50 *jh ^ cd 
1 ^ d 

4 =J ** +j § 

CO CO S , 

• 1 -j A-> ^ 

' o 3 g o 

! & g | 8 

' g 8 3 £ 

! ou 3 S g 

l| S g 8 

5 <U c OJ 

| < a ‘s | 

■ cfi 'C fi ’fl 

■ » g o £ 
1 g 3 a s 

I p m S 3 ^ 

i S.g|s 
= S g? 

I 4 ) fi U 11 


Ui 

S s 


« 3 » 

B £ 1 ? 13 
8 73 l g 
3 « 3 c 3 

CO d _( 

3 2 d •§ 

a; (U cd h 
S t 4 n d 
M ^ o >> 
Cw O ^ G 5 

s^lfl 

8 .§ J ts 

alls 


’2 3 cs 

3 S «S c 

+j <u 2 ^ 

§ a 2 *2 

^ a ^ 

<u w o cd 

^ <8 » 5 

§I||t 

&2 § g i 

0 ■on t 3 <b 

3 S %0 

S « 2^ ft 

. u a ^ -a 

■a ^ * x S 

H CO XS rj g 

c u 2 5 § 

g a 5 £ Si 

A ft id d 1 

’d «> d »H CO 

g. * s « S 
0 a 

3 a 1 •&’g u 

o V U H r* 

c « c ®5 

3 2 -g * O 
d _q cL J2 -h 

b £ v 2 B 

2 u ifl 2 <u 

3 §1* ! 

9Sljji 

1 4 -> n o 

1 c * H 

w 1 -a d S 
c/> * a 50 v 

Si i|| 

O 2; u c •g 

“ 1 1 1 s . 
t a a « 8 

Z 5 M <u ^ 
111 u V & CO 

co T5 co 
U* cd ^-t—i 5 4_> 
CC u O | C 

= s 

< § a •> u 

rn 9 cd u t-i 

y 3 33 3 

2 tjo cj 1 a; co 


MEASUREMENT SIGNAL — A representation of a measurand within a measuring system, note 
— The input to a measuring system may be called the stimulus; the output signal may be 
called the response. (vim> 
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feet applied to the measurement of voltage; (c) the Doppler effect applied to the measurement 
of velocity, ivimi 



PROBABILITY DENSITY FUNCTION (pdf) — A mathematical expression describing the func- 
tional relationship between a specific value of an attribute or variable and the probability of 
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RESPONSE TIME — The time interval between the instant when a stimulus is subjected to a 
specified abrupt change and the instant when the response reaches and remains within speci- 
fied limits of its final steady value, ivimi 
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X. being the result of the ith measurement and x being the arithmetic mean of the n results 
considered. NOTE — (1) The experimental standard deviation should not be confused with the 
population standard deviation crof a population of size N and of mean m, given by the formula: 
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UNIT (of measurement) — A specific quantity, adopted by convention, used to quantitatively ex- 
press values that have the same dimension, (van 



the electrical zero; in some types of instrument, the mechanical zero may be indeterminate. (4) 
There is also a “data zero,” e.g.; digital telemetry systems typically operate from 0 to 5 V, with 
“data zero” at 2.5 V. (vimh- 
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An effective approach to determining and implementing a limit on test process uncertainly in- 
volves defining a minimum measurement reliability target for TME parameters. In practice, 
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End-item utility is compromised and operating costs are increased if incorrect decisions are 
made during testing. The risk of making these decisions is controlled through holding TME 
uncertainties to acceptable levels. This is done by maintaining minimum levels of TME mea- 
surement reliability. These minimum levels are the measurement reliability targets. 


CRITERIA 2 — Calibration intervals lead to observed measurement reliabilities 
in close agreement with measurement reliability targets. 
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CRITERIA 4 — Calibration intervals are arrived at in the shortest possible time. 



Assume that the uncertainty growth behavior of a TME population of interest can be modeled 
by the exponential reliability model described in Section B.9: 





As Figure B. 1 shows, the calibration interval can be a sensitive function of the measurement 
reliability target. As mentioned earlier, setting an inappropriate measurement reliability target 



FIGURE B.1 — CALIBRATION INTERVAL VERSUS MEASUREMENT RELIABILITY TARGET, The re- 
lationship between calibration interval and reliability target for a TME represented by the 
exponential reliability model with R 0 = 1.0 . 
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permit this inference. Consequently, serial-number histories are accumulated typically in ho- 
mogeneous groupings, usually at the manufacturer/model level. 
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spaced closely enough to enable the detection and recording of accurate times to failure. These 
failure times are inserted into a likelihood function incorporating the probability density func- 
tion of the model of the failure-time distribution given by 
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B.7.2.3 Modified Gauss-Newton Iteration Method 

The method of getting consistent maximum-likelihood coefficient estimates is a modified 
Gauss-Newton technique. The approach uses Eq. (B.12) but departs from the method de- 
scribed in the previous section by introducing a convergence coefficient X e [0,1] as follows: 

e r+l = e r + Xb r . (B.i3) 




Hartley’s method works by using the value X mtn for X in Eq. (B. 13). Unfortunately, for multipa- 
rameter reliability models. Hartley’s method as described does not invariably lead to conver- 
gence. 
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Since yjj = y tJ , and %y g = n t R ( , Eq. (17) can be written 



The RSS, which has n-m degrees of freedom, contains the dispersion resulting from lack of fit, 
with the pure error scatter. The dispersion owed to lack of fit, called the lack-of-Jit sum of 
squares (LSS) is obtained by subtracting ESS from RSS: 



However, while rejection confidence level should surely be an important factor in the selection 
process, there are other considerations. One such consideration is the interval corresponding 
to a predicted reliability equal to the target reliability. 


a; o V 

5 || 

« 2 y 

(/) G w 
v O <L> 
•S CJ 
G o 


55 ^ 

•d cd co 

g a 13 

III 


a 43 

<D £ o 

^ C +J 

H O tUO 

s 

d 5 Oh 
£ Oh CO 


g g § 

09 fi £ 

x) « m 

pS S fi 

SSPfi 

<u 'C - 
S <u u 
^ am x 

S aJ u 
0 co ^ 

*c 1 S 

V R CO 


v .2 

a 73 *S 
0 cj £ 

id in r 

-3 y * 
5 'o 
d ^ .a 

O +J 
O <D Oh 

g >: & 

£3 <L> 

O ^ rQ 
aj o 

&73 2 

^ r; rt 
rt S 2 

SS » 

^ B § 

a " -c 

o d <u 

a 5 £ 
o a c 
c o « 
s « -3 
■g* g* S 

o rt g 

fi 5P3 


B & « 

•s c a 
S 2 cT 
o S £ 
g g 3 
8 3 a 

ill 


■ v s* m 

fi g y 3 

^ O rt n 
g I. ^ H 
k tyn 0 

?■£ j?~ 

f® 11 

n S2 ^ ^ 

cd - z? ti 

1 iD ri 

■£ 3 cd « 
0 H u -c 
^ d 

|| § s 
3 53 2 6 

*H P S 

S u 

S T3 P 
^ £3 <u . 
TJ « 0 

| o j o 

■•3 g co od 

o c <u o 
a § ^ co 

w O O +-> 

<D (M CO 

Iffil 

<n <D S 

tuo d T3 

0 5 0 O 

*d , <u 0 
S 4 h d c 
y U 2 y 

3 -a a 3 

» S o 42 . 

o & a 2 

S . p 3 

£ <l> ^ d 
^ co cu a 

gsfi 8 

g o o 

2 £3 73 -i- 1 
fi *■ -o »3 
fi -2 g - 

.22 ^ fi 
fifi fi 5 

rt H y Q 

$S$s 


G is computed for each trial reliability model: 




Then, defining a vector d with components 

d„(t,e r ) = f^!^'| , v = 1 , 2 , 3 , .... m, (B. 27 I 
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Modified Gamma Model: 


r Jh H*-) 
v d) 

I i -s 

. o £ 


‘5 CO £ 
^ 0 -) 4h 
p O XJ 

£ "O c 

•S 05 P 


si « i 

III 

1 « «■ 
gas 
£.5 o 

* £?£ 

go rl ^ 

i §j 

§ ii 
^ 8 ^ 
Shi 

i|| 

•S S c 


lir 

Hi. 


£ & &: 
(l! W § 


ii ii dT 
<QS <QJ ^ 
na 


<«& 

ii oT 
<DS ^ 
na 


ii qa* 
'Dh ^ 
<•& 


co . u 
7 " T> 5 

H 2 3 

C <U 03 

S 6 S 

&g g 

b co _. 
• O _£< T 3 

^ '5, .8 


*5 ! 5 
■0^.2 
C U « 

III 

8 '55 £ 

g 3 C 

I— 

a t * 

P <8 2 


VI j .. pj w 

■D 8 * S 

^ J a £* 

S 1 Sh 

O « § 2 

O g 81 

— o xi 13 

CO h-j c 

- ! S’s 

CD O u «3 


CL 10 -H 

3 S 8 



S s -O : 

u G ft 


« w 

O 


N || 

*2 « 

5 i 


Z%$r 

p -g £ 

fe B o 
O ^ XI 

Ilf 

u 5 CJ 1 

<D 3 ( Cd 

CL l ^ 

42 

ITXJ £ 

3 « 03 

5 3 M 

03 X! <L> 

|S a 

S 42 03 

2 -6 a 

a k. 3 ■ 

s , 

^ H CD 

S o £ 

° G E- 

CD w 

> g , 
?S:8 
M jjs 
CQ a E 
.2 

3 ttO’O ’ 

Sg-S 

= 86 
°? a § 
a t.s 

co£* 

G 73 rt 

| G g 

|«| 


+-» i O 
CO <D -M 
O *H 
o ^ 

tj 2 § . 

o. q tj 
G « g 
“ G B 

WOO , - 

a £ & 

* 5 L cd ^ 

^Z 3 CzJ ^ - 
in cd ■ 
u +j 
« G u 

T 3 O S-, 
,rH Td » 

* -2 OJ 

<2 h d 
^ <d t 1 

■s f I. 

5 § I 

8 « a 

fcl o cd 


d o g 

S ‘G *3 
s 2 tJ 

co S 3 

h +j o 

.Si co S 
dd £ d 
^ ^ o 

o CD + J 
v T3 

o H « 


£ d T3 

5 *3 <u 


a ^ a 


u «g 

B cd co 


s a I 

i|! 

E 03 to 
C co o 
£ 5 PTJ 
53 T3 g 
a i_, t> 

•a 3 a • 

G I n .2 
o G <u co 

g I s % 

to a g S 


0.5 ra 

CD SJD CO 

,o c ^ 
co ’8 o 

■a p 8 


g g •§ £ 
_§ ft g a 

pf= 3 ,i U 

!dgfl4 

.3 o &> ^ 

B fi » q 
Court 
£ 5 C TJ 
tj g .2 o 
^ £ ttf 5 

co 53 43 ^ 

G ^ S 

g S (D o 

i g » i 
rt ,pH o Z 3 

I'gsi 

li.fi 

■H cd co CD 

C H D ^ 


u u ^ 

% 5 •§ 

a 8 § 

rt ° T 3 

■a a ^ 
5 4J q 

to rt « 
8 1 * 
3 5 g 

o' d d 


o jS a ’S 

o to -G b 

'a * B ° 
2 g * S 
S 8 b.G 

--, *0 rt o 

| 2 6 5 

,S 5 P 

cs <d a y 

S S co i 


° d j 

S o; T 3 

So o 

§|S 
spI I 

13 c3 'S 
3 9) g 

a s a 

E S3 S 

s“l 

5 s l 

cd co 2 

^ o ^ 

2 'O co 

2 ^ ,pH 
cd -h p£ 

si § 
ll§ 

<D P-> CD 

a .2 a 
co Td B 
o ,pH 

S u u 

5 g £ 

V 23 X 
G "2 rt 
H G 09 

^ O _, 
CO O 
h v o 
<rt 03 ^ 

CD <D E 


Appendix B — MATHEMATICAL METHODS FOR OPTIMAL RECALL SYSTEMS 



(B.34) 



0=X ' Rj(l-R t ) ' ■ (B.39) 

x=b\ x ) 

The parameter is identified as a gem if R L > R {(t^j,6). 

By following the same treatment with “instrument class” for “manufacturer/model” and 




where (t^j and R Jfl are as defined in Eqs. (B.34) and (B.35). 

Letting K represent instruments within the TME manufacturer/model grouping of interest, the 
aggregate MTBF for the parameter is given by 
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Again, dog and gem identification at the manufacturer/model level is done by substituting 
“manufacturer /model” for “parameter” and “instrument class” for “manufacturer/model.” 
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Appendix B — MATHEMATICAL METHODS FOR OPTIMAL RECALL SYSTEMS 



M(T). Following this weeding-out process, M 0 , X, and /?are recomputed, and a more represen- 
tative picture of M(T) is obtained. Next, the slope of M(T), given by 
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The straight-line fit and the upper and lower confidence limits are determined by regression 
analysis. If calibration history data are recorded by parameter, parameter reliabilities will have 
to be converted into TME reliabilities for data truncation analysis. 



Using these quantities, a candidate identification coefficient is determined according to 

c riJ/T- n nrrr / ^nnr 



B.1 5 Establishing Measurement Reliability Targets 

Establishing measurement reliability targets involves a consideration of several trade-offs be- 
tween the desirability of controlling measurement uncertainty growth, and the cost associated 
with maintain ing such control. The trade-offs are applicable whether the goal is management 
of a ground-based calibration interval analysis system or designing TME for spaceflight appli- 
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C.2 The Test and Calibration Support Hierarchy 

Test and calibration infrastructures are characterized by several technical and management 
parameters. These parameters include calibration system, test system and end-item perfor- 
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“cost of acceptance of tested /calibrated attributes” involves a concept developed 
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suits for the entire hierarchy and detailed visibility at each level. 

A simplified dia g r am of the test and calibration support hierarchy is shown in Figure C.l. In 
the hierarchy, the end-item is placed at the top of the chain. Below the end-item is the test 
system and below the test system is a series of calibration systems, culminating in a primary 
calibration system (e.g., NIST), labeled Calibration System 1. 
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in that reported values are used as nominal values until the next calibration. Also, the lack of 
specified tolerances for calibration standards will probably be eliminated in future calibration 
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BOP Measurement Reliability — Test Process 
Accuracy 
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calibration interval. Consequently, the usage of TME attributes is referenced to average-over 
period (AOP) times. 
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TRUE EOP » TRUE (END OF PERIOD) OBSERVED EO P = PERCEIVED (END OF PERIOD) 

PERCENT IN-TOLERANCE PERCENT IN-TOLERANCE 

RELATIVE TO PERFORMANCE LIMITS RELATIVE TO TEST LIMITS 


FIGURE C.2 — THE CALIBRATION CYCLE. The elements of the calibration cycle include test 
or calibration, usage (the calibration interval), shipping and storage, data recording, and 
repair or adjustment. 







As a result of UUT testing or calibration, we “observe” a UUT EOP measurement reliability 
given by 





The true UUT measurement reliability at time of test or calibration is given by 


where 



■ 



the standard deviation due to reversion or rebound of UUT attributes away 
from values set as a result of adjustment. 



The first component of the RHS of Eq. (C. 10) is obtained using the Bayes’ relation 

/(xlnot adjust)P(not adjust) = /(not adjustlx)/(x) . (C. 1 1) 
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Exponential Model 

If the measurement reliability of an item is characterized by a constant out-of-tolerance rate, X, 
the measurement reliability in effect after an interval I is given by 

=***«"" • (C.21) 

from which 


Using Eq. (C.4) in (C.22) gives 




given by Eq. (C.4) and R . given in Eq. (C.19) or Eq. (C.20). 




C.4.2 Interval Adjustment to Tolerance Limit Changes 

An alteration of an attribute’s performance limits results in a redefinition of the standard by 
which the attribute is judged in- or out-of-tolerance. This is shown in Figure C.4. 
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To compute a new A and a new interval I' , given a performance limit change from L per to L' per , 
we imagine the following sequence of events to occur: 

m The UUT is received for calibration after an interval I. The UUT is tested and/or ad- 
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Since the calibration interval / was presumably managed to achieve a value of R obs equal to 
the desired target measurement reliability, it is assumed that the observed measurement reli- 
ability will be unchanged from its original value. Given this assumption, we have from Eq. 
(C.24), 
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P(pass) = J/(x)dx jf{y\x)dy 
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ie probability of a false alarm is given by 

P(false alarm) = P(lxl< L ,lyl> L, 




C.6 Average-Over-Period Reliability 

From Eq. (C.42), it can be seen that a viable measure of the quality of the test or calibration 



For the random walk model, there are two possibilities. The first covers cases governed by the 
adjust-always policy (L ac y=0) and the second applies to other policies. For adjust-always cases. 


(l/f)J 2F - -££L - 1 Idt , (random walk/ adjust always) (C.45) 



where R b is given in Eq. (C.19) for L adJ * 0 adjustment policies and in Eq. (C.20) for the 
L ad j 0 adjustment policy (for which Eq. (C.47) is an exact expression). Working from Eq. 
(C.30), <r can be expressed as 
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functional failures encountered during use. Since the present discussion is concerned primar- 
ily with cost and performance as affected by test and calibration, only this subset is of interest 
in the present context. To focus on this subset of repair actions, we define a parameter L rep , 



It is assumed, under ordinary circumstances, that these quantities are known. Substituting 
these variables in Eq. (C.50) and using Eqs. (C.51) and (C.52) gives 
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As Eqs. (C.48) and (C.49) show, o aop depends on or, equivalently, R^. These quantities 
are, in turn, determined by the accuracy of the test system and the quality of the test and cali- 
bration support hierarchy. 
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As stated earlier, acceptance cost applies only to the end-item. The quantities that follow, how- 
ever, apply to any UUT encountered at any level of the test and calibration support hierarc y. 
Of these, we first consider costs associated with UUT downtime. UUT downtime results in a 
requirement for replacement spares available to cover items submitted for test or calibration. 
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C.9 Multiple Product Testing 
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C.9.1 The General Multiple Testing Model 

The typical end-item multiple testing scenario uses four stages of end-item testing, as shown 
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FIGURE C.6 — THE GENERAL MULTIPLE TESTING MODEL. Items enter the testing process 
with parameter values distributed according to the pdf Jq(x). As nonconforming items 
are rejected at each step, parameter values take on distributions described by the pdfs 
/..(xlpass). Stress (thermal, vibration, etc.) is encountered by end-items within each 
stage. Following stress, parameters are distributed according to the pdfs f t {x). 









a s ^ = response of the end-item attribute to the stress applied to the end-item sub- 
population passing the jth test of the ith testing stage. 

Ji[x) = pdf for the parameter under test for the end-item subpopulation that suc- 
cessfully passes the ith testing stage. 
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Comparison of this result with Eq. (C.67) shows that the pdf for x following the first test is 
given by 
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stances, report different measured values for a given UUT attribute. Rebound error was defined 
earlier. Shipping error is an estimate of the upper limits to which the UUT attribute can be dis- 
placed because of shipping and storage. 


TABLE C.3 

Measurement Uncertainty Components 
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D.2 Computation of In-Tolerance Probabilities 

D.2.1 UUT In-Tolerance Probability 
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considered in-tolerance if e Q - L t < X t < e Q + L r In other words, populations of TME measure- 
ments are not expected to be systematically biased. This is the usual assumption made when 
TME are chosen either randomly from populations of like instruments or when no fore- 



and where 




Given these label reassignments, the needed set of accuracy ratios can be obtained using Eq. 
(D.2), i.e., 

r/= <tA / <T f ', i = l,2,-,n. 


‘inally, the tolerance limits are relabled for the UUT and the first TME according to L' 0 — L { and 





a) u 

C O <L) rH , 

OrtWJC- 

a E ^ ^ 

co E <u -j 1 
u tg ’g ^ , 

& .2 2 TS : 
.S ^ | o,j 
•g rt S -g 

§ S 2 | 

S U « § 

a -g -c ^> 

d o _ o 

-p ^ X) 7 ^ 

to ^ V ft 
d co 

•H ^ 

0 ) .2 d <u ' 

5 td S 5 

_ J-I o 
a n Vi 

1 1 S ° ; 

| s£ » 

is® C 

■h 3 ffl "cv • 

o £ hM oJD 

O g fl JJ 

J2 <2 ;p <u 
O rt 
<D u 43 
•y £ co +j 

2 43 .g rt 

s’Sci 

iiti: 

Sg-lt; 
■si f B 
* 2 1 s ; 
§1 jj g 

o a co ^ 

S <u ,S +j 

5i!l ; 

slill 

a; i d * p 

2 d C as 

3 ** -d 2 

2 a .s s 

c 3 A -2 c 
fl « 
rr m j> a 
j> a q 

5 o 2 « ■ 

2 « 8 m 

C T-l M 

■gall 


^ ~ *H 

a *5 

'O <u 

CT* d CO 


O 



D.3.2 Accounting for Bias Fluctuations 

Each attribute bias standard deviation is a component of the uncertainty in the attribute’s 
value. Bias uncertainty represents long-term growth in uncertainty about our knowledge of at- 



In these samples, let n f be the number of measurements taken using the ith TME’s attribute, 
and let X (J = Y 0 - Y y be the jth of these measurements. The sample mean and standard devia- 
tion are given in the usual way: 




bility and bias of the UUT attribute. In this, we are interested in the attribute as an entity dis- 
tinct from process uncertainties involved in its measurement. 








struments are about equal to their average-over-period values. The three instruments are 
managed to the same R* target, have the same tolerances, and are calibrated in the same way 
using the same equipment and procedures. Therefore, their standard deviations when the 
measurements were made should be about equal. According to Eq. (D.2), the dynamic accu- 
racy ratios are 


Then, by using Eq. (D.5), we get 
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- 1 + exp -In 0.72 



To compute the in-tolerance probability for TME 1 (instrument 2), the UUT and TME 1 swap 
roles. By using the transformations of Section D.2.2, we have 


in place of X± and X 2 in Eq. (D.5). Recalling that <j' 0 = a 0 in this example gives 




Thus, by Eq. (D.3), the in-tolerance probability for TME 2 (instrument 3) i 



Since the components of X are s-independent, we can write 

/(Xle 0 ) = /(X 1 le 0 )/(X 2 le 0 )-"/(X rl le 0 ) , (D.17) 

where 
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The pdf ./(X) is obtained by integrating Eq. (D.19) over all values of e Q . To simplify the nota 
tion, we define 
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If wanted, these bias estimates could serve as correction factors for the three instruments. If 
used in this way, the quantity 7 would be added to all measurements made with instrument 1 . 
The quantity 1 would be added to all measurements made with instrument 2. And, the quan- 
tity 8 would be subtracted from all measurements made with instrument 3. 





Although the coefficients c ( , i = 1,2, ••• , n, are in the strictest sense random variables to a first 
approximation, they can be considered fixed coefficients of the variables X. . Since these vari- 
ables are normally distributed (see Eq. (D.18)), the variables y/ t are also normally distributed. 
The proper expression is 
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To illustrate the determination of bias confidence limits, we again turn to the proficiency audit 
example. In this example. 
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FIGURE E.1 — BASIC MEASUREMENT MODEL. Measured values are responses of measure- 
ment systems to measurand values. Responses of individual system stages are a function 
of a set of characteristic parameters, the measurand value, and the values of other system 
responses. 
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E.2 Measurement Error Modeling 

The output y(Y|x) of the measurement system differs from the measurand by 

e(Y|x) = y(Y|x) - x . 





FIGURE E.3 — TWO-STAGE SERIES SYSTEM. The output y is a measurement of the input X. 

The error in y is a function of the errors of the responses of the stages Sj and S 2 . 

Denoting the ideal, error-free first-stage output as Y° and assuming zero measurand error, 
we write 


= P n * + Pi 2 

= [Pn + £ {Pn)] x + P12 + £ (Pi 2 ) 

= (PiV + Pi° 2 ) + ^(Pl 1) + £ (Pi 2 ) 
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E.2.2 Series-Parallel Systems 
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where Y g is the jth input to the ith stage. Consider the three-stage system in Figure E.4. It’s 
easy to show that, with appropriate notation, the error in the output of each stage can be wnt- 


N 

w 


e 

ci 

i-H 

II 





o; g 


» S * 
® o ft 

flj Q, CO 

o q £ 

d o a; 

O CD *-* 

K 4 h s 
c3 ° aj 

co £ § 
0) O O 

CO Tj ° 

G a -r 
O C g 

co 4 h -£ 




Appendix E - ERROR ANALYSIS METHODS 245 




Substituting this approximation in the expression for Y ( gives 
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E.3 Small Error Theory 

Consider the output of a stage S f , given an input Y . . If the stage response is characterized by a 
mathematical function/ and a set of parameters p„, j = 1, 2, •••, m ( , then in general, 
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Substituting in Eq. (E.14), the first-order Taylor series expansion error equation becomes 
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To explore in detail the issue of methodological relevance, it is helpful to review some back- 
ground on why measurements are made and how analyzing uncertainty leads to understand- 
ing, interpreting, and managing measurement results. 
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knowingly encountering attribute values associated with negative consequences, we say that 
our knowledge of the attribute's value is characterized by high levels of measurement uncer- 
tainty. If the reverse is the case, we say that measurement uncertainty is not significant. 



(1) Lack of an Uncertainty Model — The first drawback involves the failure to gauge the relative 
impact of each component of error on total uncertainty. Some error components may con- 
tribute more significantly than others. Without an uncertainty model, based on a rigorous er- 
ror model, arbitrary and unwieldy weighting schemes tend to be used whose applicability is 
often questionable. 
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From Figure F.3, it is evident that although the pretest population is normally distributed, the 
post-test distribution of product attribute values is nonnormal. Accordingly, treating posttest 
product attribute values as being normally distributed could lead to erroneous inferences 
about their uncertainty. 13 
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Substituting these pdfs in Eq. (F.4), 
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In Eq. (F.7), the subscripts ms, e, and s refer to “measuring system,” “environment,” and 
“setup”, respectively. Measurement system and environmental process errors are broken down 
into a bias (b) and a precision error (e). Setup error is conceived as constituting a bias only: 
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F.3.3 Accounting for Perception Error 

The operator’s perception of a measuring system result is usually subject to error. Perception 
errors arise in a number of ways. For example, in reading an analog meter, errors due to dis- 
crepancies between the operator's vantage point and the nominal meter-reading position may 
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The proliferation of desktop computing capability throughout industry has removed the pri- 
mary obstacle to implementing complex mathematical methods in the work environment. 
Hence, there are no overriding practical reasons why the practical method cannot be put to 
use by scientific and engineering personnel. Some additional work is required, however, to 
bring this to fruition. Future efforts are principally needed in the areas of error-model devel- 
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F.4.1 .2 Determining the Expectation Value for the Measurement 
Results pdf 

The pdf f(y\x) is found by averaging the error sources in Eq. (F.ll) over their respective dis- 
tributions. 
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For normally distributed measurand values, the a priori pdf /( x) is (assuming zero population 
bias) 
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Using this expression with Eq. (F.21) in Eq. (F. 15) gives the expected distribution of measured 
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The measuring system readout informs the operator that the measurement result is some- 
where between z-p k and with uniform probability. The conditional distribution that 

applies to this uniformly distributed perception error is 
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Comparing Eq. (F.32) with Eq. (F.25) shows that if even a single error source is nonnormal, the 
resultant pdf may be substantially different in character than if all sources are normally dis- 
tributed. 
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F.5.2 Estimating Measurand Values 

In making measurements, we are often primarily interested in ascertaining an estimate of the 
value of the measurand and in obtaining some confidence that this estimate is sufficiently ac- 
curate to suit our purposes. Extension of the foregoing methodology allows this objective. 




If the variable A denotes the acceptable (in-tolerance) range of attribute values, and its com- 
plement A denotes the corresponding range of out-of-tolerance values, then consumer risk 
(CR) and producer risk (PR) are calculated according to 
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F.5.5.1 Measurement Confidence Limits 

Substitution of Eq. (F.20) in Eqs. (F.34) and (F.35) gives the (1 - a) X 100% confidence limits 
for observed measurement z : 


or, alternatively, 




It should be pointed out that the process of estimating a maximum-likelihood value for an at- 
tribute involves both measuring the attribute and making a priori statements about its distri- 
bution. If, in the development of Eq. (F.25), a nonzero mean value had been specified in the a 



The component parts of these relations are easily calculated. From Eq. (F.23), 


where a z is defined in Eq. (F.24). From Eq. (F.20), the joint probability for both z and x lying 
within A is given by 




F.5.6 Example — s-independent Error Sources with Mixed 



G dH/(uk C Pl ^ /(z|u)+ t*p» du/(u|x Ct <fe/(z|i ' ) 




F.5.6.2 Measurand Bias Estimate 
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F.5.6.3 Measurand Confidence Limits 


Upper and lower confidence limits are calculated for this example by numerically or graph- 
ically solving the following expressions for L x and L 2 : 
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In developing expressions for measurement uncertainty for use in measurement-decision risk 
analysis, it is evident that simply quantifying an overall system standard deviation is not suf- 
ficient. Instead, a mathematical expression of the statistical error distribution is required. The 
development of such distributions is described in Appendix F. 
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ment is unknown. This makes projections of risk or other variables by which measurement er- 
ror can be managed all but impossible. 
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Later, the methodologies for developing system error models and uncertainty models will be 
described. These methodologies provide a framework by which measurement system errors 
and uncertainties can be identified, estimated, and analyzed. 


G.2 Identifying the Measurement System Errors 

The figure below shows a temperature measurement system. Following the prescription de- 
scribed in Section 4.4, the analysis of the measurement uncertainty of this system involves the 
development of a system error model. The development of this model will trace the measured 
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Measurand Fluctuations — If the measured value is a quantity that will be communicated for 
use in some practical application, random fluctuations that cause deviations from this re- 
ported value are a source of error. Randomly occurring differences in measurand value should 
not be confused with any time- varying aspect of a measurand, such as its signal frequency. 
Measurand fluctuations are unknown and undesirable phenomena that randomly alter mea- 






from which the voltage difference AV = V C -V A is expressed as 

AV = (a 0 - b 0 ) + (a, - b 1 )AT + (a 2 - b 2 )(AT) 2 + (a 3 - b 3 )(AT) 3 + 
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Nonlinearity — The response of a filter over the range from its cutoff frequency (j c ) to its ter- 
minating frequency ( f n ) is usually considered to be linear. Departures from this assumed lin- 
earity constitute errors. 



Gain is the ratio of the amplifier output signal voltage to the input signal voltage. Gain 
ire those that lead to a uniform shift in expected amplifier output vs actual output, 
rors are composed of inherent (bias) errors and temperature-induced (precision and 




Offset — Offset voltages and currents are applied to the amplifier input terminals to compen- 
sate for systematically unbalanced input stages. 
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G.2.7. Sampling (A/D Converter) 
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Noise — As expected, stray voltages are sensed along with the signal voltage and contribute to 
the voltage level applied to the network. In addition, thermal fluctuations in components cause 
fluctuations in voltage drops. 



where m is some integer determined by the sensing function of the A/D converter. As wall be 
discussed later, the uncertainty associated with each step is one-half the value of the magni- 
tude of the step. Consequently, the uncertainty inherent in quantizing a voltage 
V is (1 / 2 )(a / 2 n j , or A / 2 n+1 . This is embodied in the expression 


G.2.10. Data Reduction and Analysis (Data Processor) 

The quantized output from the A/D converter is input to a data processor. Since the output is 
digital, the interface between the A/D converter and the data processor will be assumed not to 
constitute an error source. The data processor converts the binary coded number to a value 
and applies any correction factors that may be appropriate. 




For example, suppose that the series is truncated to second order. Then the data processor 
solution for the temperature difference becomes 
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while the decimal encoded display is presented in steps of size 


This means that a binary encoding of a voltage V into a representation V' = 2 x h b will be trans- 
lated into a decimal representation ^" = 10^^, where x and y are integers. The quantization 
error that results from expressing an analog value first as a binary coded value and second as 
a decimal coded value is the sum of these two values: 

v A S/2 

Quantization error = ±(h b ±h d )/2 = ± — • 
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G.4 Methodology for Developing a Measurement 
System Error Model 


♦ 



MEASURAND 

VALUE 


FIGURE G.2 — THE MEASUREMENT MODEL. The output of the 8th stage is a function of the 
parameters of the stage and of the input vector X = (x,Y 3 , Y 6 ). 

The output of the measurement system, denoted y( Y|x) , differs from the measurand by an 
error: 





Sensitivity (temperature to voltage) 
Thermocouple/reference junction hysteresis 







If measurand fluctuations are not a factor, then eW can be set to zero and 



The input to the low-pass filter from interface 1 is 

Y 2={ 1 + P22){ 1 + P2l) Y l + P23 • 

Using the general model described earlier, the error in Y 2 is found to be 




Low-Pass Filter (V 3 ) — The parameters are 





+ y 2 ° e (p 31 ) f*fc 



The input to the amplifier from interface 2 is 


The error in this input is 




The error in the amplifier output is 
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p g3 = Correction factor (applied as a compensation for losses and gains in the 
signal path) 

The output of the measurement system is given by 




G.6 Methodology for Developing a Measurement 
System Uncertainty Model 
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Developing the uncertainty model from the error model involves first writing an expression for 
the statistical variance of the error in v. It will be worthwhile to pause here and discuss some 
of the properties of variances. 




var(ax + by) = a 2 var(x) + b 2 var(y) . 

Applying this rule to the error model of the velocity measurement example above gives 
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G. 7.2 Interface 1 (Reference Junction— Low- Pass Filter) 

The uncertainty in the signal-passed input to the low-pass filter is given by 


VI 


VI 

VI 


*■? 

Al 


CO 
CM 
CM a 

b 


CM 
CM 
CM a 

b 


CM 
CM Q, 

^ b ^ 

V 


CM . 





II II 


S a< 

II I! 


H CM CO 

CM CM CM 

b b b 


v 

u 

<D 

I 


0 


(/) 

CO 

CO 

CL 


Jo 

b 

j> 

tJO 

«) 

•rH 

Jh 

V 


CO 

§ 

a 

i 

V 

5 


I 

§ 

V 

5 

B 

§ 

i 

ts 

v 



CM 

CO 

cm a, 
b 


CM 

^ o1 

l 


i 

*■? 


CO 

cm a 
b 

CM 

%? 


CM CM 

b 



b n 


CM 

CO 

a 


& 

i 


tj 

H & 


g 8 

2 o 

■9 c 

CO f-1 

b 3 

8 .2 

a o 

CO b 

^ u 

o V 


fo CL >-< 

II II II 


0 

V+— 

Ql 

E 

< 

0 


0 

0 

0 

a 

i 

o 


CM 

0 

O 

0 


r— I CM 0 v 

CO CO -i— 1 1 >> 

6 a b-%r = f 


CO 

a 

b 

b 

<L> 

1-1 

<u 

^r 

a 

b 

b 

H) 

V 

u 

0) 

CD 

<D 

£ 

f 

0 

£ 

1 


f 


<D 

o 

a 

b 

CO 

CO 

<u 

'C 

<u 


* 

b 

■a 


1 1 1 


o 
b 
b 

U <U 
y co 

b ° 

:q a; 
13 y 

cO 

CO 

co h 

p 2 


£ o 

II II 


* 

S 

iio 

co 


a , a 


CM CO 


^ o 


CO 


_ a vr 

b b b S* 


•9 


> 

v 




VI 

'-I 


X 

rH 

£ 


*-? 

VI 

'"■J 

VI 


Al 


V* 


<n 


0 

Q. 

£ 

< 


LO 


O 


CO 

in 

sa 

m n< 

u 

b 

+ 

<i> 

,»rH 

♦tH 

<N ^ 

b 

S' 




ii ii 


^(NCO^tlOCDN 

in in in m 10 in in 

o, n, a, n, o, a 

b b b b b b 




<D 

U 

<D 

f 


<N 3 * 

b 

(N 

t? 

I 

E-t 


CM ^ 

b 

(N 




H M CO 

CD CD CD 

a ft, 

b b b 


y v 
> 

SO S 

O M 


1 ^ 
.22 p 


0 

■c 

0 

> 

C 

o 

O 

o 

D) 

C 

Q. 

£ 

0 

CD 


N- 

6 


- g 
b 8 


TJ 


\ C;; 


m 

ay 

M- 



verter stage. Uncertainties due to sampling rate, aperture time, and quantization are more elu- 
sive and may require some extra computation. 



The uncertainty of the output of the data processor is the uncertainty in the measurement sys- 
tem. If we can make the accuracy claims that were made in the discussion on data reduction 
error, then this uncertainty can be 'written 





bandwidth of 10 Hz is on the order of 10 -21 V — clearly negligible. As for noise induced by stray 
electromagnetic signals, at an upper frequency of 10 Hz, the noise generated by these is also 
negligible. 
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Suppose that we have a specification for the nonlinearity of the filter of ±0.15% of input signal 
level. From discussions with the manufacturer, we determine that these are 95% confidence 
limits for the filter at the upper end of our frequency range (i.e., 10 Hz). We again assume a 
normal distribution with zero mean for these errors and consult a table of normal deviates. 
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= Offset uncertainty 
= Amplifier nonlinearity uncertainty 



p 55 (Amplifier nonlinearity error) = ±0.02% 

p 56 (Common mode rejection error) = ±0.002% of common mode input 20 

p 57 (Amplifier noise level) = ±2.5 pV 

Common mode voltage = 10 pV (maximum) 

k (Thermal gain coefficient) = 2% /°C 



For this example, we will assume that a = 0 . Then 
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G.8.6 Interface 3 (Amplifier — A/D Converter) 

The uncertainty of the input to the A/D converter is 


cr p6i = Interface loss uncertainty 

a = A/D input crosstalk uncertainty 
"62 

<r p63 = Interface noise uncertainty 

r / 3 » y 4 °-p 51 vf=p 5lPll x 

Assume the following for the interface: 






G.8.7 Sampling (A/D Converter) 

The expression for sampling uncertainty is 
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Putting the numbers together yields the output of the D/A converter as 

Y ? = (0.995)Y 6 = 22,460gV , 
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Interface 1 



Recall that the correction will be in the digital rather than analog domain. Thus a quantization 
error component must be added to this analog error. Using the result 

Quantization uncertainty = (l00/2 14+1 )/ V3jUV = 1.76x10 4 /xV 


obtained in the analysis of A/D conversion uncertainty gives 













Before leaving this example, it is worthwhile to make a few observations about the bias and 
precision parts of the total uncertainty. The precision uncertainty s sample is a quantity that 
depends on the stability of the system under short-term environmental and other stresses and 







H.2 SI Units 

Three classes of units were established, base units , supplemental units , and derived units . The 
system is coherent; that is, all units derived from base units have the implied multiplier of one 
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Derived units are expressed algebraically in terms of base units by the mathematical symbols 
of multiplication and division. Because the system is coherent, the product or quotient of any 
two quantities is the unit of the resulting quantity. Table H.3 gives several examples of derived 
units expressed exclusively in base units. 
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H.2.4 Other Units 
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H.3 Units in Temporary Use 

In those fields where their use is well established, the units in Table H.7 are acceptable, sub- 
ject to future review. These units should not be introduced where they are not presently, in 







(c) The solidus must not be repeated on the same line unless ambiguity is avoided by 
parentheses. In complicated cases, negative exponents or parentheses should be 
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Definitive discussions of prefix rules and the use of exponents are found in ANSI/IEEE Std 
268-1982, NBS Special Publication 330 (1986 edition), and ISO Standard Handbook 2 (1982 
edition). All three of these documents are revised occasionally and the most recent versions 
take precedence. 
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Calibration and Maintenance of Test and Measuring Equipment,” Encyclopedia of 
l sics, Vol. 3, VCH Publishers, Inc., 1992. 
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Taylor, J., Fundamentals of Measurement Error, Neff Instrument Corporation, Monrovia, CA, 
1988. 

Wise, J.H., Aliased Data : Description and Avoidance Methods, Measurement Technology Center 
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